MARKED-UP TRANSLATION OF INTERNATIONAL APPLICATION 

[Description lTITLE OF THE INVENTION 

METHOD AND DEVICE FOR INPUTTING CONTROL INFORMATION IN COMPUTER 

SYSTEMS 

[0001] This application is based on and hereby claims priority to PCT Application No. 
PCT/EPOO/05917 filed on June 26. 2000. and European Apolicatiori No. 991 12287.0 filed June 
25. 1999 in Europe, the contents of which are hereby incorporated by reference. 

BACKGROUND OF THE INVENTION 

[0002] The inyention relates to a method and a device for inputting control information In 
computer systems, and in particular to an input device and a pertaining method for operating a 
mobile multimedia communication terminal (mobile videophone). 

[0003] Mobile communication terminals such as mobile phones for example increasingly have 
access to "online services". In particular the introduction of the future broadband UMTS mobile 
communication network will enable high-quality mobile Intemet access. 

[0004] Likewise, ultra-small computer systems (such as palmtop computers for example) on 
which a wide variety of organizer functions (such as address management, diary, task list, etc. 
for example) are implemented are becoming increasingly important. 

[0005] The operation of these display-driven computer systems requires an input device with 
which the information displayed on a display unit can be controlled or selected. 

[0006] Mobile ultra-small computer systems of this type usually have so-called "touch 
screens" as an input device, wherein the display unit is equipped with a touch-sensitive layer 
and it is accordingly possible either to input control information or select the displayed 
infomnation by [means of] an additional input stylus. The dfeadvantage of such [conventional] 
input devices however is that the additional input stylus has to be accommodated in the device 
and consequently may get lost. Moreover, user Interfaces for these [conventional] "touch- 
screen" display units are mechanically not very robust. 

[0007] So-called [conventional] "mouse" input devices are also known, wherein a rolling ball is 
converted into a corresponding electrical signal which is displayed as a pointer symbol or 
selection information on a display unit. In addition, control keys can be used as an input device 
for computer systems. 
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[0008] Particularly in the case of ultra-small computer systems however, input devices of this 
type are not possible because of the greater amount of space required. 

[0009] As an alternative to the touch-controlled input devices described above, so-called non- 
contact input devices are known in which, for example, a video camera records a position of an 
input stylus, evaluates it, and on the basis of the evaluation a pointer symbol or selection 
infonmation displayed on the display unit is subsequently displaced (e.g. US 5,726,685). 
Furthermore, a so-called gesture input device is known from the publication US 5,617,312 in 
which the gestures of a person are recorded by a camera and are subsequently converted into 
con-esponding control signals for controlling a pointer symbol or selection infomiation displayed 
on the display unit. However, since said input devices are based on relatively complex pattern 
recognition, they are usually prone to errors and require large amounts of computing power. 

[0010] Known from the publication EP-A-773 494 [forming the preamble of claim 1] is an 
input device for computer systems having a recording unit for serially recording image 
information; an image evaluation unit for evaluating the recorded image information and 
detemiining control infomiation; a data processing unit for processing the control infomnation in 
selection infomiation; and a display unit for displaying the selection infomiation. The image 
evaluation unit detemiines the control information from a relative displacement of the serially 
recorded image information. 

[0011] A device and a method for controlling a display is also known from publication 

JP 10 240 436 A, in which a picture or menu displayed on the display is displaced vertically or 

horizontally respectively depending on a relative movement or displacement of a CCD camera. 

SUMMARY OF THE INVENTION 

[0012] In contrast, [thel one possible object of the invention. is to create an input device and a 
pertaining method for computer systems which is space-savingrcost-effective and reliable. 
[According to the invention, this object is achieved by the"features of claim 1 with respect to the 
device and by the measures of claim 17 with respect to the method.] 

[0013] Particularly on mobile videophones, the use of an image evaluation unit in particular, 
which uses an image compression device for generating motion vectors, with the control 
information being derived from a main motion vector, enables the recording unit already present 
as well as the associated image compression method to be used simultaneously as an input 
device. 
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[00141 Respective parallel displacement components of the image information in the x and y 
directions are preferably used to generate the x and y components of the control infonnation. 
which enables control of a pointer symbol or selection infomriation to be realized on a two- 
dimensional display. The control information may however also have a z component which is 
produced from a concentric displacement of the image information in the x and y directions, 
which enables control of a pointer symbol or selection infonnation to be realized on a three- 
dimensional or quasi three-dimensional display. In addition, this enables an intuitive 
enlargement or reduction of the display scale. To produce such a concentric displacement of 
image information for three-dimensional input, the recording unit can preferably have a zoom 
function unit. 

[0015] A mobile multimedia communication terminal is preferably used as the computer system, 
with the recording unit in a mobile videophone being used both for recording a call partner and 
as an input device. Analogously, a display unit is used both for displaying a call partner and for 
displaying an Internet page, organizer page, etc. An extremely cost-effective and space-saving 
realization of the input device is thus achieved. The recording unit comprises in particular a 
CCD or CMOS sensor camera, which reduces the space requirement still further. 

[00161 The recording unit preferably records image infonnation for input control in the display 
direction of the display unit, but it can also record image information in the opposite direction. 
This is particularly advantageous when the input device is provided in a mobile videophone, 
because when the videophone is placed on a suitable surface, by moving the videophone to 
and fro, an associated pointer symbol can be con-espondingly displaced on the display unit in 
the same way as with a [conventional] mouse input device. 

[0017] The recording unit preferably has a macro camera system for recording image 
information from a very close distance, which enables, for example, an input to be made by 
placing a finger directly on the recording unit. If the recording unit additionally has a pressure 
sensor for confirming displayed selection information, then the input device can be operated in 
the same way as a [conventional] touch-saeen input device. [Further advantageous 
embodiments of the invention are characterized in the subclaims.] 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0018] [The invention is described in greater detail below on the basis of exemplary 
embodiments and with reference to the drawing, in whichl These and other obiects and 
advantaoes of the present invention will become more apparent and more readilv appreciated 
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from the followino description of the preferred embodiments, taken in conjunction with the 
accompanying drawings of which : 

[Figure]Fi£u 1 shows a schematic diagram of the input device according to one aspect of 
the invention; 

fFlQure lFig. 2 shows a schematic diagram of serially recorded image Infomiation to 
illustrate three-dimensional control; 

rPiguresl Fias. 3a and 3b show schematic diagrams of a mobile communication terminal 
having an input device according to a first exemplary embodiment; 

[FigureJEig. 4 shows a schematic diagram of a mobile communication terminal having an 
input device according to a second exemplary embodiment; and 

rFiquresl Fiqs. 5a to 5c show schematic diagrams of parts of a mobile communication 
tenninal having an input device according to a third exemplary embodiment. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

[0019] Reference will now be made in detail to the preferred embodiments of the present 
invention, examples of which are Illustrated in the accompanying drawings, wherein like 
reference numerals refer to like elements throughout. 

[0020] fFigurelFiq. 1 shows a schematic diagram of an input device [according to the invention]. 
In [FigurelFtg. 1, reference numeral 1 denotes a recording unit for serially recording image 
information, wherein first image information A essentially comprises a landscape with a house, 
two trees and an industrial building. After swiveling the recording unit 1 in the negative y 
direction, second image infomiation A' is recorded, which essentially comprises the house, the 
tree and the industrial building of the first image infomiation A plus an additional fence. Owing to 
the swiveling of the recording unit 1 , the additional tree present in the first image infomiation A 
is not present in the second image Information A'. 

[0021] An image evaluation unit 3 (AW) serves to evaluate the serially recorded first and 
second image infomiation A and A', with control information x, y and z being determined. 
Evaluation of the image evaluation unit 3 is essentially based here on determining a relative 
displacement between the serially recorded image information, i.e. between the first image 
information A and the second image infomiation A'. To be more precise, a main displacement 
vector HV which corresponds to a displacement of the main image components within image 
information A and A' is determined. 
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[0022] Given the swiveling of the recording unit 1 in the y direction illustrated in [Figurej Fig. 1, 
when the image information A and A' is evaluated in the evaluation unit 3, the main 
displacement vector HV is accordingly determined, which essentially represents a displacement 
in the y direction. The control information in the y direction thus obtained is subsequently 
processed in a data processing unit 4 (DV)in such a way that a display unit 2 connected to the 
data processing unit 4 causes a con-esponding displacement of selection information P 
displayed on the display unit 2 to P'. In rPigurelFig. 1, the selection information displayed on the 
display unit 2 is a pointer symbol P, which is located in an upper position for a first camera 
position for recording the first image infonnation A, and moves to a lower position (P*) when the 
recording unit 1 is swiveled to record the second image information A". 

[0023] In accordance with [FigureJFig^ 1 , therefore, a relative displacement of serially recorded 
image infomiation can be produced and evaluated by swiveling a recording unit 1. which 
information produces a displacement of the displayed pointer symbols P to P' on the display unit 
2. 

[0024] The relative displacement is detemnined here on the basis of the principal structures of 
image information A and A*. To be more precise, the relative displacement is not detemiined on 
the basis of micro displacements, but rather on the basis of macro displacements in the image 
information A and A'. Macro displacements are understood to be, for example, displacements of 
the general background of the image information A and A', while micro displacements are 
displacements of limited image areas as a result of quick and instantaneous movements. 
Accordingly, a car moving through the landscape illustrated in [FigurelFig. 1 would not cause a 
macro displacement, but only a micro displacement of image infonmation which is ignored by the 
input device. 

[0025] [According to the invention, onl vlOnlv macro displacements of the recorded image 
information are taken into account for the inputting of control information or the control of 
selection infomiation P and P displayed on the display unit 2, with a threshold decision element 
(not shown) setting an absolute magnitude for the macro displacement. Accordingly, a macro 
displacement is taken into account by the evaluation unit 3 if, for example, at least 90% of the 
image information recorded by the recording unit 1 exhibits the same or a similar displacement. 
A parallel displacement in essentially the same x and y direction is interpreted here as the same 
displacement, while a displacement in the z direction is interpreted as a similar displacement. 
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[0026] [Figure]F!g^ 2 shows a schematic diagram for illustrating a relative displacement of 
image information in a z direction. According to fFigurelFig. 2, the image infomnation recorded 
by the recording unit 1 comprises a house which is initially at a distance (image information A) 
and the same house which, as a result of coming closer in the z direction, is displayed as 
enlarged by the image information A*. 

[0027] Accordingly, given a displacement in the z direction, the displacement vectors are not 
ordered parallel to one another In the x or y direction, but exhibit a concentric displacement 
direction. Such concentrically oriented displacement vectors can therefore be evaluated by the 
evaluation unit 3 as control Informatidn in the z direction, thus enabling even three-dimensional 
control of the selection infomiation displayed on the display unit 2. i.e. pointer symbols P and P. 
The display unit 2 may also be a three-dimensional display unit here (e.g. holographic display 
unit). 

[0028] As an altemative to moving the recording unit 1 in the z direction, i.e. moving closer to 
the object to be recorded, the recording unit 1 can also have an (electronic or mechanical) zoom 
function unit (not shown) with which it is likewise also possible to perfonm a concentric 
displacement of the image information. The recording unit 1 may also have a rotating or 
swiveling unit (not shown) with which the macro displacements in the x and y directions can be 
realized. 

[0029] f Figures! Figs. 3a and 3b show schematic diagrams of a mobile multimedia 
communication terminal having the input device according to a first exemplary embodiment of 
the invention. In FFigure lFig. 3a, MM denotes a mobile multimedia communication terminal, as 
may be realized for example in the form of a mobile videophone with Internet access. In addition 
to the usual function units, the mobile multimedia communication terminal MM essentially has a 
recording unit 1, a display unit 2 and an activation button TTor activating/deactivating the input 
device. In the case of an Internet access, the enlarged Internet page IS (web page) is displayed 
on the display unit 2 for example, with a pointer symbol P being displayed as selection 
information at the position I. To move the pointer symbol P from position I to position II, 
according to rFlgurelFig. 3b a user can, for example, move the mobile multimedia 
communication tenminal MM downward in front of his face, with the result that, instead of image 
infomiation A at position I, the image infomiation A' at position II is now recorded by the 
recording unit 1. The resulting macro displacements are detected by the evaluation unit 3, as 
described above, and are processed by the data processing unit 4 in such a way that the 
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display unit 2 now displays the pointer P at position II of the displayed Internet page IS. A 
displacement of the pointer symbol P showing the selection infomriation to P' is thus achieved in 
this way by displacing the recording unit 1 relative to the recorded background. 

[0030] As was already described above, it Is also possible to perfomi a swiveling instead of 
the vertical displacement of the mobile multimedia communication terminal from a position I to a 
position II, which enables the pointer symbol P to be displaced in the same manner to position II 
for the pointer symbol P'. 

[0031] Mobile multimedia communication terminals, as illustrated by way of example in 
rFlquresl Figs. 3a and 3b, are especially advantageous if they are equipped as mobile 
videophones. Mobile videophones already have a large display unit 2 as well as a recording unit 
1 for recording image information. Moreover, such mobile videophones already have an image 
compression device as image evaluation unit 3 for compressing the recorded image information. 
The algorithms used for image compression detect a movement in the image information A and 
A\ with the movement of the total image infomriation being calculated in such a way to produce 
a total or main motion vector which is equivalent to the movement or displacement of the mobile 
multimedia communication terminal MM or the associated image infomriation A and A' 
respectively. This total or main motion vector, which can be readily calculated during image 
compression, can now be processed as control infonnation with con^esponding x, y and z 
components, and displace the selection information, i.e. the pointer symbols P or P', 
accordingly. Since the compression algorithms used are essentially already present in a mobile 
videophone, no additional costs are incun^ed. To switch over or activate/deactivate the input 
device, the mobile videophone or mobile multimedia communication tenninal MM requires only 
an activation button T. 

[0032] [Figure]Fiq. 4 shows a schematic diagram of a mobile multimedia communication 
tenninal having an input device according to a second exemplary embodiment. In [Figure]Fig^ 4 
the same reference symbols denote the same or con^esponding components as in rFigures ]Fiqs. 
3a and 3b. so a detailed description is dispensed with below. 

[0033] On a mobile multimedia communication terminal MM having an input device according to 
the second exemplary embodiment, the display unit 2 does not display the complete Internet 
page IS, but only a selected section F. This way of displaying of information, for example from 
the Internet or from another application, has the advantage that the infomriation on the displayed 
page IS is more readily legible for the user. 
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[0034] Accx)rding to the [present inventionl second exemplary embodiment , the input device now 
has intuitive control of the information page IS to be displayed. To be more precise, a user need 
only displace the mobile multimedia communication terminal with the recording unit 1 in the x 
and y directions in order to reach the respective sections of the information page IS. so that the 
information page IS can be read intuitively (i.e. by displacing a viewing window) without using 
any further keys or input media. 

[0035] In addition, a displacement in the z direction, i.e. a forward/backward movement of the 
mobile multimedia communication terminal MM, can enlarge or reduce the size of the selected 
section F. which enables Intuitive use comparable to the use of a magnifying glass. 

[0036] According to rFiqures lFias. 3 and 4, the recording unit 1 is located on the front of the 
mobile multimedia communication terminal MM. It can however also be located on the rear of 
the temiinal. or a second recording unit may be arranged on the rear of the mobile multimedia 
communication terminal MM. Given suitable realization of the optics, i.e. the ability to focus the 
recording unit 1 even at a very short distance, the mobile multimedia communication terminal 
MM can also be placed rear down on a suitably structured surface and the whole unit can be 
displaced in the same way as a mouse. The recording unit located on the rear detects here, as 
described above, the relative movement or relative displacement of the recorded image 
information (structured surface) and determines from it the con*esponding control infomriation for 
controlling the selection infomriation, i.e. a pointer symbol P or a display section F of the 
information page IS. This results in the input device being particularly easy to operate, since it 
essentially corresponds to a [conventional] mouse input device. In this case the mobile 
multimedia communication terminal preferably has the selection buttons (not shown) known 
from [conventional] mouse input devices. 

[0037] [FigureslFigs. 5a to 5c show enlarged partial views'^of a mobile multimedia 
communication temninal MM having an input device according to a third exemplary embodiment. 
In [Figures! Figs. 5a to 5c. the same reference symbols denote the same components as in 
[Figures] Figs. 1 to 4, so a detailed description is dispensed with below. 

[0038] According to [Figures! Figs. 5a to 5c, the input device according to the third exemplary 
embodiment can be used in the same way as a [conventional] so-called "touch-pad" input 
device. In [FigurejRg. 5a, reference numeral 1 again denotes a recording unit 1, but one which 
is a special macro camera system. To be more precise, the recording unit 1 according to 
fFiguresl Figs. 5a to 5c is able to record very fine structures in the immediate vicinity of the 
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camera, by [means of] which, for example, the finger lines of a finger placed directly on the 
recording unit are recorded in sharp focus. 

[0039] According to f Figures] Figs. 5a to 5c, the display unit 2 shows a telephone directory list 
with a plurality of names and conresponding phone numbers. The telephone directory list 
displayed on the display unit 2 thus con-esponds to an infomiation page IS, with the selection 
information being represented neither by a pointer symbol P for indicating a selection position, 
nor by a display area F, but by a predetermined selection field B in the form of a, for example, 
colored bar. To control the input, or to select the name Meier with phone number 345678, the 
user then places his finger on the recording unit 1 , with the input device activated by [means of] 
the activation button T (not shown). According to fFioure lFig. 5b, the finger lines are now 
recorded as image infomnation A, and the user moves his finger downward to displace the bar 
B. [Figure] Fig. 5c shows here a partial view of the mobile multimedia communication terminal 
MM, with the finger shown at a lower position. The recording unit 1 accordingly detected a 
displacement of the recorded Image information A -» A' on the basis of the finger lines and 
evaluated this as a vertical displacement downward. The evaluation unit 3 accordingly outputs 
corresponding control infomnation to the data processing unit 4, which results in the bar B being 
displaced downward on the display unit 2 and now highlighting the desired name Meier with 
phone number 345678. 

[0040] To select the desired name, according to the third exemplary embodiment the input 
device can also have a pressure sensor for confinning the displayed selection information, i.e. 
the name highlighted by the bar B, which dispenses with the need for further fixed keys or so- 
called menu-driven "soft keys". 

[0041] Given suitable dimensioning of the recording unit 1 and the evaluation unit 3, in this way 
it is also possible to make an input in a way known fi-om [conventional] "touch pads" and/or 
"touch screens" for example. The integration of a pressure sensor in the recording unit 1 further 
reduces the space required for the input device and enhances ease of operation. In comparison 
with [conventionaljlsnown "touch pads" in particular, with the input device according to the third 
exemplary embodiment, it is not the main focus of a surface area which is determined, but 
rather the exact position or movement of the finger is tracked on the basis of the lines present 
on it. The precision of this input device is therefore much greater than that of 
[conventionalljcDOM! input devices. With the input device according to the third exemplary 
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embodiment, therefore, essentially a relative macro displacement of the recorded image 
information is also used to control selection information. 

[0042] A CCD or CMOS sensor camera is preferably used as the recording unit 1. The 
recording unit 1 is located here either on the front or on the rear of an input or computer system 
respectively, but it is not limited thereto and may also be anranged at a different position and be 
moved around, for example by [means of| minrors, in the same or opposite direction to the 
display direction of the display unit 2. 

[0043] The invention was described above with reference to a mobile multimedia 
communication terminal. It is however not limited thereto, and likewise encompasses ultra-small 
computer systems without a communication link, such as palmtop computers for example. 
Likewise, the present invention may also be used on corded telecommunication terminals and 
other computer systems. 

[0044] The invention has been described in detail with particular reference to preferred 
embodiments thereof and examples, but it will be understood that variations and modifications 
can be effected within the spirit and scope of the invention. 



